Skip to content

Add hash → block number index + lookup API#72

Draft
mo4islona wants to merge 2 commits into
masterfrom
block-hash-index
Draft

Add hash → block number index + lookup API#72
mo4islona wants to merge 2 commits into
masterfrom
block-hash-index

Conversation

@mo4islona

@mo4islona mo4islona commented May 29, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Summary

Lets consumers resolve a bare block hash to its block number. Today the hash is only buried inside each chunk's Arrow hash column, so the only way to look it up is a linear scan (~0.1–0.6s, and worse as retention grows). This adds a proper index for an O(1) point lookup, plus an HTTP endpoint to use it.

The motivation: services that receive a raw hash (from logs, fork events, external sources) need to map it back to a block number to keep working with the standard /stream APIs.

What's new

  • Storage — a RocksDB index mapping hash → block number per dataset, kept up to date automatically as chunks are ingested, forked, and pruned. Enabled for EVM datasets for now (easy to extend to other chains later).
  • Hotblocks — GET /datasets/{id}/hashes/{hash}/block, returning {number, hash} or 404 if the hash isn't found.

Rollout notes

  • No backfill: the index covers only data ingested after this ships. Hashes from chunks that already exist at upgrade time resolve to 404 until those chunks roll off via retention. Worth calling out in release notes.
  • Safe to roll back: older binaries simply ignore the new column family.

Testing

Unit tests in sqd-storage cover ingest, fork, retention, dataset deletion, and confirm compaction leaves the index intact. Manual HTTP smoke against a live hotblocks instance still recommended.

🤖 Generated with Claude Code

@mo4islona @claude
Add hash -> block number index + lookup API
ae9883d 
New CF_BLOCK_HASHES maps dataset_id||hash to a block number for sub-ms
lookups, updated on ingest/fork/retention/delete (EVM-only), untouched by
compaction. delete_dataset now uses per-chunk txs to bound memory. Adds
find_block_by_hash and GET /datasets/{id}/hashes/{hash}/block. No backfill:
old chunks 404 until they age out.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@mo4islona @claude
delete_dataset: restore atomicity via index range-delete
99cccd0 
Clear the dataset's whole hash index with one range tombstone over its CF_BLOCK_HASHES prefix, then delete all chunk metadata and the label in a single transaction. The range-delete stays outside the tx (RocksDB forbids delete_range inside a transaction); a crash between the two leaves chunks without index entries, which 404 until re-indexed, not corruption.

Keeps memory bounded (no delete_cf per block) while making the metadata deletion all-or-nothing again, and re-takes the optimistic label lock. Adds BlockHashIndexKey::dataset_range for the prefix bounds.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@elina-chertova

elina-chertova commented Jul 5, 2026

Copy link
Copy Markdown

Heads-up: dogfooding this branch as data-hotblocks:pr-72 is degrading a shared hotblocks instance

The pr-72 build of this branch has been running on the network-hotblocks-mainnet-internal hotblocks-db since ~2026-06-03, and it has been progressively degrading every dataset on that instance (not just EVM ones):

  • Processing time (head_monitor_result_ms_last, 3h) spikes across all datasets together — solana-mainnet 404 s, berachain 103 s, linea 94 s, ethereum-mainnet 92 s — while the same datasets on the public deployment (image b5b8886, i.e. without this branch) stay <500 ms.
  • Downstream this trips Portal_Hotblocks_Data_Lag_Critical / Avg_Processing_Time_High for solana-mainnet, solana-devnet and hyperliquid-fills (see subsquid/infra#544).

Why non-EVM datasets are hit by an EVM-only feature: index_block_hashes() (added to insert_chunk/insert_fork in write/dataset_update.rs + write/tx.rs) runs inline inside the ingest transaction and, for every EVM chunk, iterates every block's hash into CF_BLOCK_HASHES via self.transaction.put_cf. Even though is_indexed_kind short-circuits non-EVM datasets, the ~25 EVM datasets' extra per-chunk index writes + the new column family's compaction contend on the shared RocksDB instance, starving the highest-throughput consumers (solana most of all). The pressure grows with retention, which is why it took weeks to become alert-worthy.

Not asking to change the feature's design — just flagging that before this ships to a shared multi-dataset instance, the hash indexing likely needs to move off the synchronous ingest path (background/async index maintenance) or otherwise be prevented from contending with unrelated datasets' ingestion. Immediate mitigation is rolling the internal image back to b5b8886 (subsquid/infra#544).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@mo4islona @elina-chertova